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ABSTRACT 



This study invesit igated possible reasons for the low 
performance on the text -based writing assessment of the Delaware Student 
Testing Program (DSTP) in 2000, especially for grades 3 and 5, and considered 
ways to improve classroom instruction. In the first part of the study, a 
panel of teachers reviewed the anchor papers from the assessment and the 
process of testing. Panel members rescored the anchor papers of a given grade 
independently and then worked in a small group to discuss and finalize their 
scores. In the second part of the study, a second panel of teachers 
participated in a rescoring session for a sample of 100 text-based writings 
per grade, scoring students' writings holistically and analytically using the 
five-point scoring rubric. Both panels also reviewed the directions for 
administering the test and test booklets. Results of the study are reported 
in the categories of: (1) the process of testing; (2) construct validity 

evidence; (3) text -based writing scoring; (4) text -based writing development; 
and (5) text-based writing instruction. In each of these categories, some 
possible explanations for student performance were identified, including the 
introduction of a new text-based writing task at each grade level in 2000, as 
exemplified by the change from persuasive writing to informative writing at 
grade 3 . The paper cautions that the study was done rapidly and that much 
additional research is needed, focusing on the stability of scores over time 
and the processes of reader training and scoring. Attachments contain the 
scoring rubric, instructions for the anchor paper review, rescoring 
directions, a list of panel members, correlation matrices, and records for 
the rescoring. (Contains 14 tables, 4 figures, and 8 references.) (SLD) 
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Report on a Special Writing Study 



Introduction 

The objective of the Delaware Student Testing Program (DSTP) is to measure student 
progress toward the Delaware Content Standards. Each spring, all public school students 
in grades 3, 5, 8, and 10 take the statewide assessment in reading, writing, and 
mathematics. The writing assessment consists of a text-based writing task and a stand- 
alone writing prompt. The text-based writing task links to a passage in the DSTP reading 
assessment and students' responses to this task are scored twice, once for a reading score 
and once for a writing score. Both stand-alone and text-based writings are untimed 
Students usually take approximately 2 hours, including a 30 minute pre-writing session, 
to develop, organize, draft, and finalize their stand-alone writings. Only the final draft of 
this prompt is scored. " 

A 5-point scoring rubric (Please see Attachment A) is used to score both the text-based 
mid stand-alone responses. One reader scores the text-based writing; two readers score 
the stand-alone writing. The lowest score for the text-based writing is 1 and the highest 
possible score is 5; the lowest score for the stand-alone writing is 2 and the highest 
possible score is 10. The total writing raw score is the sum of the text-based writing 
score and the stand-alone writing score with the lowest score of 3 and the highest 
possible score of 15. & 

Over the past three years, the overall writing scores have declined in grades 3 and 5, " 
remained steady in grade 8, and increased slightly in grade 10 (See Tables la - Id). The 
average performance on the stand-alone writing shows a consistent pattern of increase 
across years for students in grades 8 and 10; minor fluctuations over time for students in 
grades 3 and 5. Student performance on the text-based writing, however, dropped to the 
lowest level in 2000 for all grades except grade 10, where the average scores dropped bv 
.66 from 1999 and .63 from 1998 to 2000 in grade 3; dropped by .59 from 1999 and .76 
from 1998 to 2000 in grade 5; and dropped by .28 from 1999 and .45 from 1998 to 2000 
in grade 8. Because of the drop in text-based writing scores, the Assessment . and 
Analysis Group decided to conduct a special writing study to investigate the possible 
reasons for the low performance in 2000, especially in grades 3 and 5. 

Purpose of the Study 

The primary purposes of this study were (1) to investigate the possible reasons for the 
low performance on the text-based writing in 2000, especially in grades 3 and 5 and (2) 
to investigate ways to improve classroom instruction in writing. 

Methods of the Study 

Ge r^l Design Due to the time constraints and the availability of information/data this 
study focused on the following five aspects: 
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• Review the test process (i.e., review of test administration and testing materials); 

• Review text-based writing scores (i.e., review anchor papers and re-score a 
sample of students' responses to the text-based writing task, and compare the 
statistics of text-based writing scores from the field test with and the 2000 DSTP); 

• Examine construct validity evidence (i.e., review available data and conduct 
additional statistical analyses); 

• Make recommendations for the development of text-based writing tasks; and 

• Make recommendations on ways to improve classroom instruction in writing. 

This study included two parts. In Part One, a panel of teachers reviewed the anchor 
papers and the process of testing (See Attachment B). Anchor papers are a sample of 
students' writings that are used as benchmarks in scoring. Each anchor paper represents a 
score point and usually represents the upper and lower levels within each score point. In 
this study, the panel members re-scored the anchor papers of a given grade 
independently, and then worked in a small group to discuss and finalize their scores. In 
Part Two, a second panel of teachers participated in a re-scoring session for a sample of 
100 text-based writings per grade. They scored students' writings holistically and 
analytically using the 5-point scoring rubric. Each writing sample was evaluated by up to 
5 teachers. Then, the panel members discussed related issues in test administration, test 
development, scoring, and classroom instruction (See Attachment C). 

Sample of Student Writings A random sample of 100 student responses to the text-based 
writing task was selected from the population of each grade for re-scoring in this study. 

■■ Panels. q[Teachers_ Two panels of teachers were invited to participate in this study, one 
for anchor paper review and one for the re-scoring session (See Attachment D). These 
teachers were selected based on their expertise in writing, teaching experience, 
experience in the development of writing assessment and scoring, familiarity with the 
Delaware Content Standards in English language arts and the writing scoring rubric, 
geographic location, and availability. 

The Anchor Paper Review Panel consisted of 9 members, including 7 teachers (78%) 
from 7 school districts and 2 staff members from the Department of Education. Among 
them, 7 are females and 2 males. Seven of the panel members (78%) have served on the 
test development committees and 2 (22%) were involved in the anchor paper pulling for 
the 2000 DSTP writing assessment. 

The Re-Scoring Panel included 22 members, 20 of them were teachers (91%) from 12 
school districts, 1 from the University of Delaware, and 2 from the Department of 
Education. Forty -one percent of the members (9) have served, on the test development 
committees and nearly half (about a quarter) were involved in the anchor paper pulling 
for the 2000 DSTP writing assessment. 

Data Analysis qnd_ Summary of Comment s To investigate the possible reasons for the low 
performance on the text-based writing in 2000, teachers reviewed, discussed, and made 
recommendations for improving test administration, test development, scoring, and text- 
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based writing instruction. All comments and subjective evaluations are summarized in 
attachments. These comments are reported in the section of the “Results of the Study”. 

The results of data analyses are presented in tables and charts. Data analyses for this 
study include: 

• Three-year comparisons of students' writing scores by grade 

• Three-year comparisons of statistics of writing scores by gender and grade 

• Correlation analysis of all types, of writing scores and reading scores bv grade 
for 2000 DSTP 

• Comparisons of means and frequency distributions of text-based scores 
between the 2000 DSTP and field test by grade 

• Comparisons of means and frequency distributions of text-based scores from 
the field test, the 2000 DSTP, and re-scoring 

• The discrepancy of text-based scores from the 2000 DSTP and re-scoring 

• The reliability indicators of re-scoring the text-based writing 

Results of the Study 

The results of the study are reported in five categories, the process of testing, construct 
validity evidence, text-based writing scoring, text-based writing development and text- 
based writing instruction. 

Process of Testing Both panels reviewed the 2000 DSTP Directions for Administering 
the Test and test booklets and compared those directions with the previous years' testing 
materials and the process of testing. Comments on the process of testing focus on the 
following issues: 

1 . Two text-based writing prompts, one for field test and one for operational test, 
should not be given on the same day, especially for younger students. 

2. The text-based writing should be given in the beginning of the reading test rather 
than as the last item of the day. 

3. The instructions for the text-based writing should be written to draw students' 
attention, such as bolded , for emphasis, using separate pages to ensure that 
students understand this item will be scored twice for both reading and writing. 

4. The text-based writing task should be formatted similar to the stand-alone 
writing prompt, such as using pre-writing. 

Review of Text-based Writing Scores 

Construct Validity Evidence The statistics of the three writing scores, text-based, stand- 
alone, and the writing, total raw scores are compared by gender and grade for 1998, 1999, 
and 2000 (Table 2). The data show that female students consistently outperformed male 
students on both text-based and stand-alone writings across all four grades and three 
years. A slight, but consistent decrease of gender differences in text-based scores for 
students in grade 3 is observed (Diff score=.30 in 1998; .22 in 1999; .13 in 2000) from 
the current analysis. 
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As indicated earlier in this report, the text-based writing tasks attach to a passage in the 
DSTP reading assessment. This passage includes several multiple-choice (MC) and 
constructed-response (CR) items. The student's response to one of the CR items was 
scored as part of the reading score using the reading scoring rubric and the text-based 
writing score using the writing rubric. 

The multiple-choice item has a stem asking a question with four choices, from which 
students select the best answer to the question. Multiple-choice items are scored 
dichotomously. Constructed-response items include short answer and extended 
constructed-response items. Short answer items allow students to make decisions by 
constructing brief responses that demonstrate the students' understanding of the text. 
Short answer items are scores on a 0-2 scale. Extended constructed-response items allow 
students to make decisions by constructing more lengthy responses that demonstrate the 
students' understanding of the text and require students to provide justification for their 
responses. The extended constructed-response items are scores on a 0-4 scale. The item 
used as the text-based writing task is an extended constructed-response item. 

Tables 3a and 3b present the correlation coefficients among five reading scores, three 
writing scores, and the SAT9 reading comprehension test scores by grade from the 2000 
DSTP. The analyses are based on the following eight variables: 

• MCITEM: The multiple-choice item score is the sum of scores on all MC items 
attached to the reading passage 

• CRITEM: The constructed-response item score is the sum of scores on all 

constructed-response items attached to the reading passage 

• PASSAGE: The passage score is the sum of scores on all MC and CR items 
attached to the reading passage 

• IREADING: The reading item score is the score on the extended constructed- 
response item attached to the reading passage that was used as the text-based 
writing task 

• TEXT: The text-based writing score is the score on the extended constructed- 
response item attached to the reading passage that was used as the text-based 
writing task 

• PROMPT : The writing score on the stand-alone writing prompt 

• WRITING: The total writing raw score that is the sum of the text-based and 
stand-alone writing scores 

• READING: The DSTP reading score 

• SAT9: The reading score on the 30-item SAT9 reading comprehension test 

The results show that the correlation coefficients between the text-based writing scores 
(TEXT) and the item reading scores (IREADING) from the same CR items are .22 for 
grade 3, .45 for grade 5, .57 for grade 8, and .60 for grade 10. First, the statistics indicate 
a grade pattern, from the lowest value of the correlation coefficient in grade 3 to the 
highest value in grade 1 0. Second, the low correlation in grade 3 suggests that only 5% 
of the variance from one score associates with the other score; in grade 5, about 20% of 
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the variance from one score associates with the other score. The correlations between the 
text-based writing scores (TEXT) and the scores on the MC items (MCITEM) and the CR 
items (CRITEM) from the reading passage, and the passage scores (PASSAGE) are .19, 
.31 and .31 in grade 3, which is the lowest among the four grades. Again, the low 
correlation in grade 3 suggests that only 4% to 10% of the variance of text-based writing 
scores is associated with the MC item scores, CR item scores, and the passage scores, 
respectively. Similarly, the correlations between the text-based writing score and the 
scores on MC items (MCITEM) and CR items (CRITEM) from the reading passage, and 
the score of the reading passage (PASSAGE) are .26 to .44 in grade 5, which indicate that 
1 % to 19% of the variance from the text-based writing scores can be accounted for by the 
scores from reading. Statistics from the current analysis suggest that the text-based 
writing score is somewhat independent of the reading passage, especially in grades 3 and 

The correlations between the. text-based writing (TEXT) and stand-alone writing score 
(PROMPT) range from .36, .41, .41, and .48 for grades 3, 5, 8, and 10, respectively. A 
grade pattern is observed, where the correlation coefficient for grade 3 is the lowest 
among the four grades. The low and moderately low correlations across grades suggest 
that about 13% to 23% of the variance from the text-based writing scores is associated 
with the stand-alone writing scores. Statistics appear to suggest that the text-based 

writing measures different types of writing skills or different constructs from the stand- 
alone writing. 

The correlations between SAT9 reading scores, a standardized test, and the DSTP reading 
scores (READING) are stable across grades, ranging from .84 to .86, and no grade 
pattern is found. Moreover, the sizes of the correlation coefficients between the SAT9 
reading and the stand-alone writing scores (PROMPT) are very close, ranging from .41 to 
48 across grades without a grade pattern. The correlations between the SAT9 reading 
and the text-based writing scores (TEXT), however, show a grade pattern with the lowest 
coefficient in grade 3 (r=.33) and the highest coefficient in grades 8 and 10 (r=.48). The 
statistics indicate that there may be more measurement errors involving in the text-based 
writing scores than that in the stand-alone writing scores, especially in the lower grades. 

Ifie correlation matrix among different types of writing scores and reading scores for the 
1998 and 1999 DSTP provides additional information for the construct validity (See 
Attachment D). The correlation coefficients between reading and writing scores are 
consistent in 1998 and 1999. The correlations between text-based writing and reading 
scores are higher in 1998 and 1999 (r= .56 in 1998 and r=.60 in 1999 for grade 3; r=.60 in 
1998 and r=.56 in 1999 for grade 5) than that in 2000 (r=.33 in grade 3; r=.44 in grade 5). 

In grade 3, the correlation between text-based and stand-alone writings is lower in 2000 
(r=.36) than the previous years (r=45 in 1998; r=.46 in 1999). The correlation between 
text-based writing and reading scores also shows the lowest value in 2000 for grade 3 
(r=.63 in 1998; r=.68 in 1999; r=.53 in 2000). Such variations of the statistics across 
years of testing may be due to one or more of the following reasons- 
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• Low generalization of writing scores across topics, the purposes of writing tasks, 
and occasions; 

• More errors in text-based writing than standard-alone writing in 2000 than the 
previous years; 

• More errors in scoring text-based writing than in scoring stand-alone writing 
because of using one reader; 

• Variations in the characteristics of the reading passages and attached items from 
year to year and from grade to grade; or 

• Variations in writing skills among student populations from year to year. 

Review of Text-Based Writing Scores 

Re^scoring Anchoi r Papery To examine the accuracy of scoring, the anchor papers were 
reviewed and re-scored by the first panel. Anchor papers are typical writing samples that 
represent each score point and used as benchmarks in reader training and scoring. 
Usually, writing samples are selected to represent the upper and lower levels of each 
score point to facilitate the process of scoring. The panel members reviewed, ranked, and 
re-scored each anchor paper independently. Then, they , worked in small groups to 
discuss their scores in order to achieve an agreement, if possible. The mean of the 4-5 
new scores from re-scoring process were calculated and compared with the original 
scores. If the different score between the new score and the original score (Diff = re- 
score - original score) was equal or greater than a half point (Diff = >.5), this paper was 
flagged as inconsistency in scoring. The results of re-scoring anchor papers show (See 
Attachment F) that the new scores and the original scores are highly consistent in grades 
3 and 5 (92%), and moderately high in grade 8 (86%) and grade 10 (73%). According to 
the panel recommendation, three papers should not be used as anchor papers in the future. 

Comparing Test_ Statistics The mean and the relative frequency distributions of text-based 
writing scores from the field test and the 2000 DSTP were compared. The discrepancies 
of scores summarized in Table 4 show that the average scores from the field test are 
consistently higher than the average scores from the 2000 DSTP across the four grades, 
where the different scores (2000 DSTP score - field test score) are -1.03, -.68, -.26, and - 
.65 for grades 3, 5, 8, and 10, respectively. Chart 1 illustrates the discrepancy between 
the relative frequency distributions of the text-based scores from the two occasions. In 
grade 8, the relative frequency distributions of the writing scores from the. field test and 
the DSTP 2000 are nearly identical with the same mode above the 2-point on the score 
scale. The difference in mean scores is due to 15% more students receiving a 2-point and 
11% less students , receiving a 3-point in the 2000 DSTP than that in the field test. 
Similarly in grade 10, because of the mode of 3-point from the field test and 2-point from 
the 2000 DSTP, the frequency distribution of the 2000 DSTP shift to left from the 
frequency distribution of the .field test with a lower average score. In grade 3, the 
frequency distribution of the field test scores approaches a normal distribution with the 
mode of 3-point; while the frequency distribution of the 2000 DSTP scores is positively 
skewed with the mode of 1 -point, which is 2 score points lower than the mode of the 
score distribution from the field test. In grade 5, the mode of the writing scores is 3-point 
of the field test scores with a nearly normal frequency distribution; while the mode is 2- 
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point of the 2000 DSTP scores with a positively skewed frequency distribution. To 
examine the differences between the two score distributions from the field test and the 
2000 DSTP, Chi-Square Goodness of Fit test, a non-parametric test, was conducted (See 
Table 4b). The results indicate that the relative frequency distributions of text-based 
writing scores from the two occasions are statistically significantly different at the p< .00 
level for all the four grades. 

Re-scoring Text-Based Writing Samples To further investigate the possible reasons that 
caused the low performance in text-based writing, a re-scoring session was arranged. A 
sample of 100 students' responses to the text-based task was randomly selected from the 
population of each grade. The panel members reviewed and re-scored the sample of 
writings independently. Each paper was evaluated holistically and analytically using the 
same 5-point scoring rubric. The analytic scores were only used as evidence to support 
their professional judgments on the quality of student writing. They worked in small 
groups to discuss their initial scores and assigned final scores (See Attachment F). 
Agreement among panel members was not required. Since the anchor papers were under 
review, the process of re-scoring was conducted without anchor papers. The records of 
re-scoring from the panel members can be found in Attachment G. 

The different text-based writing scores (Diff = re-score - 2000 DSTP score) between the 
2000 DSTP and re-scoring by the panel were calculated based on the following criteria: 

• Majority rule was applied to decide the final re-score in the process of re- 
scoring. 

• If the same number of readers agreed and the same number of readers did not 
agree with the 2000 DSTP score, the 2000 DSTP was used as the final re-score. 

• If only one reader reviewed the paper, this reader's score was used as the final 
re-score. 

Tables 5a and 5b present the summary of the discrepancies in scoring by grade. Data 
shows that the average agreement between the two scoring process is 65% across grades, 
58% in grade 3, 66% in grade 5, 48% in grade 8, and 47% in grade 10 (Table 5b). Two 
patterns are observed: First, the scores assigned in the 2000 DSTP are consistently lower . 
than that in re-scoring across the four grades; second, the distributions of the different 
scores are negatively skewed, which indicate more papers were scored lower than higher 
in the 2000 DSTP. On the average, 35% (1 13 papers) of the papers were scored 1 -point 
lower in the 2000 DSTP than that in re-scoring, 8% (27 papers) of the papers were scored 
2-points lower, and less than 1% (1 paper) of the papers were 3-points lower than they 
should be according to the panel's judgment. There are 9% (28 papers) of the papers that 
were scored 1 -point higher and less than 1% (2 papers) of the papers scored 2-points 
higher than they should be across grades according to re-scoring. Grade by grade, it is 
found that 37% of the papers were scored lower and 5% of the papers scored higher in 
grade 3 from the 2000 DSTP; 29% of the papers were scored lower and 5% of the papers 
were scored higher based on re-scoring in grade 5. The data show that 46% of the papers 
were scored lower and 5% of the papers were scored higher in grade 8; 37% of the papers 
were scored lower and 16% of the papers were scored higher in the 2000 DSTP. 
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Tables 6a and 6b list the mean and frequency distributions of text-based scores from three 
occasions, field test, 2000 DSTP, and re-scoring. The data indicate that the writing 
scores re-assigned in this study are, on the average, higher than the 2000 DSTP, but lower 
than the field test in grades 3, 5, and 10. In both grades 3 and 10, the average re-score is 
closer to the 2000 DSTP in absolute value (.27 for grade 3; .16 for grade 10) than to the 
field test results (.76 for grade 3; .49 for grade 10). In grade 8, however, the average re- 
score is .38-point higher than the field test and .64-point (about one standard deviation) 
higher than the 2000 DSTP. 

To examine how reliable the re-scoring process was, the number of readers used to re- 
score each paper and the degree of agreement among readers were analyzed. The 
majority of the papers were reviewed and re-scored by at least 3 readers, 96% in grade 3, 
71% in grade 5, 51% in grade 8, and 68% in grade 10. In grades 3, 5, and 8, less than 5% 
of the papers were re-scored by a single reader; in grade 10, 22% of the papers were re- 
scored by one reader, which may be due to the content and the length of student writings 
in higher grade. In addition, the number of readers who assigned the same score that 
differs from the 2000 DSTP scores is used as the indicator of reliability of scoring (Table 
7). The data indicate that the majority of the different scores were assigned by the panel 
in this study based on the agreement among 3 to 5 readers, 95% in grade 3, 70% in grade 
5, 61% in grade 8, and 69% in grade 10. Fifteen percent of the papers (26 papers) 
receiving a different score were based on one reader; but most of them are 1 -point higher 
rather than lower than the 2000 DSTP scores. 

The panel members were also asked to make professional judgments on the difficulty 
level of the 1999 and 2000 text-based writing tasks by reviewing the reading passage and 
attached items. The results are summarized in Attachment F. 

Text-Based Writing Development During group discussion, teachers provided comments 
and suggestions related to the development of the text-based writing. Their comments 
focused on three major issues: passage selection, wording of the prompt, and use of the 
writing rubric (See Attachment I). 

• Passage Selection: Passages should be engaging and the difficulty level should be 
consistent from year to year. Third grade teachers preferred realistic stories as the 
basis for the text-based writing. Fifth grade teachers thought passages should be 
infomiative selections dealing with social studies or science. 

• Wording of the Prompt: The wording in the prompt should always direct the 
students back to the text so that information from the text is included in the 
response. “Use details from the text to support your answer,” should be in all 
prompts. Students should understand the concepts implied in the wording of the 
prompt. Developers should take care in using “user accessible” language in 
writing the prompts. 
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• Use of the Writing Rubric: Teachers discussed the possibility of adapting the 
general writing rubric so that each text-based writing item would have an item 
specific writing rubric. With specific guidelines provided in these item specific 
rubrics, it would be easier for scorers to determine when a student's writing is off 
topic. 

Text-bused Writing Instruction Teachers' comments related to instructional issues of the 
text-based writing focused on professional development in crafting text-based prompts 
and on identifying a variety of reading passages with which to write such prompts. They 
emphasized the need to have students write in response to a variety of text types (literary, 
informative, and technical) across content areas, and for teachers to model the process of 
making connections to the text and pulling out relevant details (See Attachment I). 

• Tenth grade teachers pointed out that most of the writing done by high school 
students is text-based, and that text-based writing is not a separate type of writing. 
Written responses to texts are produced as forms of persuasive, expressive, or 
informative writing. High school teachers also expressed a concern regarding 
block scheduling, where students may have only five weeks of instruction prior to 
the administration of the DSTP. Finally, tenth grade teachers suggested that high 
school English teachers have gone away from literary analysis in lieu of an 
emphasis on stand-alone writing prompts, which may sacrifice students’ writing 
in response to text. 



* _ Fifth grade teachers stressed the importance of students making connections with 
characters in a story. They suggested that grade-level ‘teams or district 
committees (lead by reading cadre representatives) develop questioning activities 
for teachers to use to improve students’ performance on text-based writing. They 
also pointed out the need to release sample student responses to text-based writing 
prompts. b 

Limitations of the Study 



As indicated in the beginning of this report, the current study was designed and 
conducted based on the available data within a short period of time. Due to the 
limitations of the study, the author suggests cautions in reviewing, interpreting, and using 
the results of this study. 6 

• Information, such as sampling procedures and students' scores on the field test, is 
not available for review and additional analysis. 

• Even though the sample of student text-based writings was randomly selected, the 
small sample size, only 1% of the grade population used in re-scoring, may not 
accurately reflect the characteristics of the population because of sampling errors. 
In addition, since the anchor papers were under review, the re-scoring process was 
conducted without using anchor papers. 

• It is very important to note that previous studies have shown that the 
generalization of writing performance is low across the purpose (or discourse) of 
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writing tasks, writing topics, and occasions, especially when there are only a 
couple of items used in the writing assessment. In 2000, a new text-based writing 
task was introduced at each grade level, which may be one of the reasons for the 
fluctuation of the test scores. For example, third graders responded to an 
informative writing task instead of a persuasive writing task. Similarly, the fifth 
graders responded to an informative writing task in 1998 and 1999, but an 
expressive writing task in 2000. These changes could account for the low 
performance. 

• An important issue of educational measurement is reliability. Reliability of 
performance-based assessment, such as writing, is often defined by agreement of 
readers in scoring a single task given on a single occasion, called inter-reader 
reliability or inter-reader consistency. However, another component of reliability 
involves the consistency of measurement over repeated occasions given fixed 
readers is called score reliability. Findings from early studies suggest that reader 
consistency differed considerably ranging from .33 to .91 and score reliability 
ranged from .26 to .60, which was dependent upon the number of points on the 
scoring scale, rating conditions, and changes in assessment programs (Dunbar et 
al, 1991; Fitzpatrick et al, 1994). The results of an experimental study conducted 
in Virginia (Moon et al, 1996) indicate that methods used for training and scoring 
(i.e., training readers to score multiple writing prompts at a single session or 
training readers sequentially score a new writing prompt) impact both reliability 
and validity. They also found that readers scored differently using the same 
scoring method on the same set of students' papers across years. To better 
understand the nature of direct writing assessment and provide valid and reliable 
measures of student achievement, more research questions, such as the stability of 
scoring over time, the process of reader training and scoring, and score reliability 
across topics, discourses, and occasions, need to be further explored. 
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Means, Standard Deviations, and Frquency Distributions of Writing Scores by Grade and Year 
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Means and Standard Deviations of Writing Scores by Gender, Grade, and Year 
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Table 3b 

Correlation Matrix of DSTP Writing, DSTP Reading, and 
SAT9 Reading Scores by Grade 
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Table 4 

Means and Frequency Distributions of Writing Scores in Field Test and 2000 DSTP 
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Table 6a 

Summary of Discrepancies in Scoring Text-Based Writing 
from field test, 2000 DSTP, and Re-Scoring 
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Table 7 

Number of Readers Used in Re-Scoring Text-Based Writing 
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Table 8 

Reliability Indicator of Text-Based Writing Scores in Re-Scoring 
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INSTRUCTIONS FOR ANCHOR PAPER REVIEW 



1. In the folder, you will find the following: 

• Two copies of the Form of Anchor Paper Review, one for each grade 

• Non-disclosure Form 

• Writing Scoring Rubric 

• Reading passage and the Question for the text-based writing 

• Anchor papers for selected grades 

2. Procedures for Anchor Paper Review: 

Step One: (On your own) 

• Sign the Non-disclosure Form. 

• Read the Instructions for Anchor Paper. Review. 

• Read the reading passage and the question for the text-based writing. 

• Read the scoring rubric. 

Step Two: (On your own) 

• Read each anchor paper carefully. 

• Rank all the anchor papers from the lowest (as #1) to the highest (whatever 
the number applies). You may assign the same rank order to more than one 
paper, such as.two papers are ranked as #3 and the following one should be #5 
instead of #4. 

• Assign a score to each anchor paper using the 5-point rubric. You may use 

high or low to differentiate papers even though they receive the same score 
point(s). 

• Record your scores using the Form of Anchor Paper Review. Match the Code 
with the 1 -digit number sequential number on the right top of each paper and 
fill in the rank order and the score you have assigned to each paper. 

• You also need to record the 1 1 -digit paper number (located on the right 
bottom of each paper) in under the column of Paper Number of the Form of 
Anchor Paper Review. 

• Provide your comments using the space below the Form. 

• Please return the entire folder to Kim by August 7, 2000. 

Step Three: (Group meeting if necessary) 

• Have a group meeting to share your scores with other reviewers. 

• Group Discussion 

• Make your second judgment 

• Compare your judgments with the original scores. 
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Attachment C 
Procedures For Re-scoring 
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Agenda 



Study for the DSTP Writing Assessment 
8:30 am- 3 :30pm, Tatnall Building, Dover, August 17, 2000 



Welcome and Study Brief (Zhang) 8:30-8:45 

Scoring Rubrics Training 8:45-9:05 

• Holistic Scoring Rubrics for Writing (Bolig) 

0 Holistic Scoring Rubrics for Reading (Bolig) 

• Analytic Scoring Rubrics for Writing (Kelley) 

Evaluate Text-based Writing Tasks 9:05-9:30 

• First Round Evaluation 

Review and Re-score Students' Writings 9:30- 1 1 :00 

• First Round Scoring 

Small Group Discussion 11:00-12:00 

Lunch 12:00-12:40 

Continue Group Discussion and Second Round Scoring 12:40-2:00 

Evaluate Text-based Writing Tasks 2:00-2:30 

• Second Round Scoring 

Grade-Group Discussion 2:30-3:30 



• Recommendations 

• Comments 

• Evaluation 



Appendix A 

Delaware English Language Arts Content Standards 



Students in Delaware public schools, using the process of effective readers writers 
listeners, viewers, and speakers, will be able to: 5 



Standard#! 

Use written and oral English appropriate for various purposes and audiences. 

Standard #2 

Construct, examine, and extend the meaning of literacy, informative, and technical texts 
through listening, reading, and writing. 

Standard #3 

Access, organize, and evaluate information gained through listening, reading, and viewing. 

Standard #4 

rnd cStiue kn ° wledge accessed through print and visual media to connect self to society 
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Definitions of Cognitive Categories for 
DSTP Reading Comprehension Assessment 



Three stances of are used to measure the depth of reading comprehension. These stances 

are: Determining Meaning, Interpreting Meaning, and Extending Meaning. 

• Questions in the Determining Meaning stance require the reader to demonstrate an 
overall understanding of the passage. The focus is on how the reader begins to make 
meaning of the text. 

• Questions in the Interpreting Meaning stance require the reader to go beyond the initial 
understanding to develop an interpretation of the text. The reader goes beyond first 
impression to construct a more complete understanding of what has been read. 

• Questions in the Extending Meaning stance require the reader to stand apart from the 
text and critically consider it. This stance involves critical examination, evaluation, 
and analysis. 




So 



20 



Delaware Student Testing Program - Instructional Guide for Writing 

(An Analytic Adaptation of the DSTP General Rubric for Writing) 



The following characteristics determine the ggm ISSponse i# meeting the heeds of the audience and 

fulfilling the writing purpose. 



Score of 5 



Score point 5 
meets dll the 
criteria listed in 
score point 4. In 
addition, a paper 
receiving this 
score shows an 
exceptional 
awareness of 
readers ' concerns 
and, heeds. 

The student may 
have shown an 
exceptional Use 
ofi 

Development 
. strategies 
specific to the 
purpose for 
writing 

• Distinctive 
style, voice, 
tone. 

• Literary 
devices 

9 Compositional 
risks 



Score of 4 



Score of 3 



Score of 2 



Unified with Generally unified 



smooth transitions, 
a clear arid logical 
progression of 
ideas, and an 
effective 
introduction arid 



Sufficient, 
specific, arid 
relevant details 
that are fully 
elaborated. 



witli some 
transitions, a clear 
progression of 
ideas, arid an 
introduction and 
closing. 



Specific details 
but may be 
insufficient, 
irrelevant, or not 
fully elaborated. 



Minimanyunified'f 
arid may lack 
transitions or an 
introduction Of 



Score of 1: 



Lacks unity.; 



Some specific 
details but may be 
insufficient, 
irrelevant, and/or 
not elaborated. 



Consistently 
complete 
sentences with 
appropriate variety 
in length and 
structure'. 



Generally 
complete 
sentences with 
sufficient variety 
in length and 
structure. 



A consistent style Some style and 

with precise and generally precise 

vivid word choice. I word choice. 



Few, if any, errors I Some errors in 
ip standard written staridard written 
English that do not English that rarely 
iritirfere with interfere with 

understanding. understanding. 




Some sentence 
formation errors 
arid a lack of 
sentence variety. 



Sometimes general 
and repetitive 
word choice. 



Several kinds of 
errors in staridard 
written English 
that interfere with 
understanding. 



No or few specific 
details that are 
minimally 
elaborated. 



Frequent and 
severe sentence 
formation errors 
and/or a lack of 
sentence variety. 



repetitive, and/or 
confusing word 
choice. 



Frequent and 
severe errors in 
standard written 
English that 
iriterfere with 
understanding. 





Holistic vs. Analytic Scores 



There is no concrete correlation between the holistic score for a piece of writing and the 
analytic scores for each of the textual features at work within that piece of writing. A 
holistic 3 does not mean that each analytic score is also a 3. NOr does a holistic 3 mean 
that the sum of the analytic scores is 15, which when divided by 5 (the number of textual 
features) would “give us” that holistic 3 . It’ s not that simple. 

First, the textual features are not weighted equally. Generally, Organization and 
Development carry far more clout than Conventions, Sentence Formation and 
Style/Word Choice, while also considered more significant than Conventions, are not 
quite as critical to a piece as Organization and Development. Additionally, the nature of 
a particular prompt, its audience, or its purpose may subtly emphasize a particular textual 
feature arid thereby influence the scoring. 

Second, in addition to the' distinct analytic scores, a holistic score also takes into account 
how those individual textual features work together to create the whole piece of writing. 
In essence, the whole is “greater” than the sum of its parts. But “greater” does not imply 
that the holistic score is higher; it means that there is more to consider in the holistic 
score than just the analytic' Scores for each of the textual features. 

Our Task 

When analytically scoring the samples for this study, consider only the four levels below. 
Score points 4 arid 5 front the DSTP Rubric are lumped together for the purposes of this 
study. We are less concerned with determining the differences between those score 
points than we are with determining the differences, in terms Of the textual features, 
between score points 3, 2, and 1. 
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The Chicken and the Egg: Which comes first, holistic or analytic? 



What holistic score should we assign to 
analytic scores? Why? 



Example A 



Textual Feature 


Analytic 

Score 


Organization 


1/2 


Development 


1/2 


Sentence Formation 


3/4 


Style/Word Choice 


3/4 


Conventions 


3/4 



Example B 



Textual Feature 


Analytic 

Score 


Organization 


1/2 


Development 


2/3 


Sentence Formation 


■ 3/4 


Style/Word Choice 


3/4 


Conventions 


3/4 



Example C 



Textual Feature 


Analytic 

Score 


Organization 


1/2 i 


Development 


1/2 


Sentence Fonnation 


1. 2/3 


Style/W ord Choice 


2/3 


Conventions 


3/4 



paper with the following configuration of 



Example D 



Textual Feature 


Analytic 

Score 


Organization 


. 3/4“” 


Development 


3/4 


Sentence Formation 


3/4 


Style/Word Choice 


3/4 


Conventions 


3/4 



Example E 



Textual Feature 


Analytic 

Score 


Organization 


2/3 


Development 


3/4 


Sentence Formation 


2/3 


Style/Word Choice 


2/3 


Conventions 


3/4 



Example F 



Textual Feature 


Analytic 

Score 


Organization 


3/4 


Development 


3/4 


Sentence Formation 


2/3 ~| 


Style/Word Choice 


2/3 ' 


Conventions 


3/4 



ERIC 



• Begin developing a sense of the holistic score by looking at Organization and 
Development. These two textual features carry the most weight. 

• The combination of analytic scores for Sentence Formation, Style/Word Choice, 
arid Conventions may shift — -either up of down— the holistic score developed by 
considering the analytic scores for Organization and Development. This shift will 
seldom be more than one score point. 

• Conventions alorte do not make a piece of writing better, they merely make it 
correct in terms of Standard Written English. Conventional correctness does not 
improve the quality of a piece of writing that lacks unity, specific detail, sentence 
variety, style, etc. Quality comes with improvements in the other textual features. 
Conventions dp make the piece more readable, but readable writing that says 
nothing and makes no sense still says nothing and makes no sense. 



2/2 
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Anchor Paper Review Committee 
F or FY 2000 DSTP Writing Study 



Name 


Gender 


District 


School 


Test Development 




Committee 


Grade 3 & 5 










Cookie Bolig 


F 


Department of Ed. 


Assessment & Analysis 


Y 


• Jackie Shockley 


F 


Cape Henlopen 


R.A. Shields Elementary 


Y 


Marty Hodgkins 


F 


Appoquinimink 


Redding Intermediate 


Y 


Linda Mitchell 


F 


Indian River 


East Millsboro Elementary 


N 


Chris Evans 


F 


Brandywine 


Mt. Pleasant High 


Y 












Grade 8 & 10 










Mike Boyd 


M 


Lake Forest 


Lake Forest High 


Y 


Kate Szegda 


F 


Red Clay 


District Office 


Y 


Doug Grudzina 


M 


Capital 


Dover High 


N 


Mike Kelly 


M 


Department of Ed. 


Curriculum Development 


Y 




O 

me 



Anchor Paper Pulling Review Committee 
For DSTP Writing Study 



Name 


Gender 


District 


School 


Test Development 










Committee 


Grade 3 










Jackie Shockley 


F 


Cape Henlopen 


Shields Elementary 


Y 


Janice Trainer 


F 


Christina 


Wilson Elementary 


N 


Ann Whitman 


F 


Milford 


Benjamin Banneker Elementary 


N 


Nancy Carter 


F 


Milford 


Benjamin Banneker Elementary 


N 


Cookie Bolig 


F 


Department of Ed. 


Assessment & Analysis 


Y 












Grade 5 










Linda Mitchell 


F 


Indian River 


East Millsboro 


N 


Marty Hodgkins 


F 


Appoquinimink 


Redding Intermediate 


Y 


Mary Currie 


F 


Milford 


Milford Middle 


Y 


Christine Poehlmann 


F 


Appoquinimink 


Redding Intermediate 


N 


Karen Sheets 


F 


Woodbridge 


Woodbridge Elementary 


N 












Grade 8 










Kate Szegda 


F 


Red Clay 


District Office 


Y 


Ginger Angstadt 


F 


Capital 


Central Middle School 


Y 


Linda Poorman 


F 


Colonial 


Wallace Wallin 


N 


Gwnne Ash 


F 


University of DE 




N 


Denise Speicher 


F 


Indian River 


District Office 


Y 












Grade 10 










Mike Boyd 


M 1 


Lake Forest 


Lake Forest High 


Y 


Catherine Laverick 


F 1 


srccvT 


Todgson Vo-Tech 


N ' 


John Drumheller 


M ( 


Lape Henlopen ( 


Jape Henlopen High 


N 


Judy Smith 


F 1 


folytech I 


5 olytech High 


N ■ 


Pam Wilson 


F I 


5 olytech I 


5 olytech High 


N 


Doug Grudzina 


M ( 


Capital I 


Dover High 


N 


Mike Kelly 


M I 


Department of Ed. C 


Curriculum Development 


Y 
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Correlation Matrix between Reading and Writing 



1999 


READING 


TEXT-BASED 


STAND-ALONE 


WRITING 


Grade 3 










READING 


1.00 








TEXT-BASED 


0.60 


1.00 






STAND-ALONE 


0.57 


0.45 


1.00 




WRITING TOTAL 


0.68 


0.77 


0.92 


1.00 


Grade 5 










Reading 


1.00 








Text-Based 


0.56 


1.00 






Prompt 


0.55 


0.46 


1.00 




Writing 


0.64 


0.79 


0.91 


1.00 


Grade 8 










Reading 


1.00 








Text-Based 


0.65 


1.00 






Prompt 


0.55 


0.49 


1.00 




Writing 


0.68 


0.80 


0.91 


1.00 


Grade 10 










Reading 


1.00 








Text-Based 


0.55 


1.00 






Prompt 


0.59 


0.47 


1.00 




Writing 


0.66 


0.75 


0.92 


1.00 


1998 


Reading 


Text-B 


Prompt 


Writing 


Grade 3 




- 






Reading. 


1.00 








Text-Based 


0.56 


1.00 






Prompt 


0.52 


0.44 


1.00 




Writing 


0.63 


0.77 


0.91 


1.00 


Grade 5 










Reading 


1.00 








Text-Based 


0.60 


1.00 






Prompt 


0.57 


0.45 


1.00 




Writing 


0.68 


0.79 


0.90 


1.00 


Grade 8 










Reading 


1.00 








Text-Based 


0.60 


1.00 






Prompt 


0.63 


0.54 


1.00 




Writing 


0.70 


0.80 


0.94 


LOO 


Grade 10 










Reading 


1.00 








Text-Based 


0.56 


1.00 






Prompt 


0.56 


0.47 


1.00 




Writing 


0.65 


0.81 


0.90 


1.00 



* All correlation coefficients are calculated based on aggregated data. 





Attachment F 

Summary of Anchor Paper Review 



Sfc 
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Notes 


#1 & #2- Long sentence! Probably scored better for 


reading. 


#3 - Point of view few details. 






#6 - Should score low for reading. 








#10 - Original score should be a 3+ according to the 


rubric. It meets the aspects of a 3. It does not meet 
all aspects consistently for a 4. 
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Writing Task Evaluation 
Grade 3 



Comments: 



• Confusing for students. It arises that tigers don’t live in forests with foxes. The 

wording for the 1999 prompt explained it better than in the 2000 prompt did for 3 rd 
graders to understand. The 1999 story was more interesting. Children can relate to 
those pets. 

• The question doesn’t give enough stem to make students relate to the nature of the 
folk tale. 

• I think it would be very hard for a child to achieve a 4 or 5 on this task because if 
they respond to the prompt and the reading, it’s basically a retelling. There is little 
opportunity to demonstrate any writing “flair” they may have which can push a paper 
to a 4 or 5. 
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Writing Task Evaluation 
Grade 5 



Comments 



• Felt the students needed an organizer. 

• The 2000 test is harder. 
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Writing Task Evaluation 
Grade 8 



Comments 

• Students have some difficulty with “heritage”. The question implies that the 
author and narrator are the same person and they are not. 

• I’m not sure that this means anything to me, really. I felt like I was answering 
without a lot to go on. My low estimate is based on confusion based on the 
prompt. It assumes/implies that the author and the narrator are the same when we 
teach them. That is not the case. It forces the reader/writer to shift between 
viewpoint and voice, and that kind of stuff that can confuse issues. 
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Writing Task Evaluation 
Grade 10 



Comments 



• I think that the Cinderella question was not very engaging to the male 
population. 

• This is most likely an overly biased or subjective evaluation. I’m not sure I 
have a good idea of an “average” student. 

• I feel that the 2000 prompt uses language (“broad appeal”) that is confusing 
to the students. Many did not seem to understand that broad appeal meant 

popular” which influenced their answer. 



• The word choice ' in the question (2000 essay) make it confusing for the 
general student population. I would re-word “broad appeal”. The 1999 
question had more familiar wording and topic choice. The selection was more 
interesting to students. The 2000 selection was more literary, and students in 
grade 10 received less instruction in literature. 



• Perhaps the phrase “broad appeal” confuses some kids who cannot 
understand what it meant. 



• The format of the ? {Last on page, no bold in directions, no drafting} leads 
itself to the difficulty level. 



• 2000 I think that the Cinderella question was not very engaging to the 

male population. 
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Writing Study Group Discussion 
Grade 3 

) 

Text-Based Writing Development 

Look at choice of text. There is a big difference between 1999 and 2000. It is much easier 
for kids to connect to a person rather than a monkey. Prefer to see realistic stories used 
when kids have to be a Text-based writing. Authentic audience. 



The question in 2000 did not say use details and information for the text. The 1999 did not 
say that verb tense (present) may have confused them. The question doesn’t give enough 

stem to make students relate to the nature of the folk tale. 

Basically a retelling, so it is difficult for a child to achieve a 4 or 5. The question asked 
what do you see, what do you think and lots of kids responded, I see, I see, I see or I think. 
Confusion for students arises that tigers don’t live in forests with foxes. 



Text-Based Writing Score 

We found a big discrepancy between Harcourt’s and ours. They seem to have a bigger 
range for a 1. Check the discrepancies between our “3’s” and their same pieces which 
were scored “1”. 

We also had a problem with the invalids “the big rock” scored a 1, should it be a 0? 

Teachers should be given a breakdown of “the score”. This code can be done by codes. 
Not enough details for the story, etc. . . 

Text-based scoring rubric for writing be developed. How much emphasis is incorporated 
in the grading for using the text? 



Test Administration 

Certainly not 2 Text-based writing’s on one day. Not authentic. Use a realistic story, not a 
fantasy or folk tale. Word the prompt more carefully - scaffold. Set up like the prompt is 
to emphasize the importance. 



Test Development 



Writing Study Group Discussion 
Grade 3 



DSTP Manual 2000. Page 17 

The Directions include both questions 57/67 to take a 10 minute rest. Included “Dear Mrs 
Cabot” 
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Writing Study Group Discussion 
Grade 5 

Text-Based Writing Development 

Text-based writing added to get a more valid score for writing. 1 page versus 2 pages 
formatting. Change in short answer. 

Navaho selection is not as engaging. Don’t have background experience. Should be able 
to get enough from text. (But familiarity helps connect to text.) 

Do better with nature. Students connect. Both informative pieces (Science and Social 
Studies). 



Text-Based Writing Score 

Text-based writing used writing rubric. Need anchor papers. Is our scoring as good 
without anchors as theirs is? 



Test Administration 

Instructions: Developmental^ need to be aware of problems, of directions, understand 
graded for reading and writing. Amount of instruction needed to listen and to understand. 
Not doing 2 text-based writing’s on the same day. Consider doing text-based writing 
passage first of a set. Prewriting area. 



Classroom Instruction 

Develop text-based responses to go home with class take home books. Content area do 
also. Model text-base responses show, have to make connections and pull out details. 

Make connections with characters in the story. 

New reading people to look at district curriculum to find samples of things to work on. 

Grade levels / districts developing questioning activities to improve text-based writing. 
Sample papers for text-based writing. 
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Writing Study Group Discussion 
Grade 5 



Test Development 

DSTP Test 2000 - Student Book 22 - Response booklet 12-13 Use 2 full pages. 1999 
only 1 page. 



Grade 5 directions read much better than grade 3. Especially need to look at breaks in 
grade 3. 
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Writing Study Group Discussion 
Grade 8 



Text-Based Writing Development 

350461 - Issues: Is it on topic? Should it be scored? 

143951 — Topic issues. Is this scorable? 

11306480 - Tuff to decide between 2+/3. 

193408 - 2/3 — 

758424 - Off topic? 

721301 - Is this off topic? 

What is the purpose of text-based writing, is it to test for ability to write an audience 
purpose? “It didn’t count”. 

1. When is a piece off topic? {and/or invalid}. Is a summary that doesn’t relate back to 

the question scorable? When does 1999 tell us? 

2. How did media information one week prior to test impact scores? 

3. Stand-alone started in 90 text-based writing. . . 98 not as much experience. 

4. Author/narrator: Confusion or attention to author craft? 

5. Did kids understand the concept of heritage? 



Text-Based Writing Score 

Does a narrative text lead itself to a persuasive prompt? 
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Writing Study Group Discussion 
Grade 10 



Text-Based Writing Development 

1999 Text-based writing prompt is more opinion based and more authentic. 

2000 prompt has less room for creativity. (Harcourt wrote it). 

More like a SOAP prompt. (Subject O and purpose) 

Use “user accessible” language 

Doug says there should be more bone oriented Civil War passages. 

Text-Based Writing Score 

Relation to off topic. Should the use of the word “Cinderella” earn credit? (Re-examine 
the “invalids”). Insufficient information. 

0-1 distinction (May be something that changes per prompt). 

Test Administration 

Prompt set up alone. Not the last to do in the day. Bold it . Emphasize it’s importance. 

Separate page. Bolded for emphasize. Physically make it look like the stand alone 
prompt. 

For Special Ed. — In unclear directions, can there be another version (allowable 
modification) of the directions? 

Classroom Instruction 

Research and Instruction - When appropriate to paraphrase/quote from a source. 

T - have gone away from literary analysis due to emphasis on stand-alone prompts. 
Variety of selections in the classrooms. 

Block schedule issues (5 weeks till DSTP) 
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Writing Study Group Discussion 
Grade 10 

Text-based writing is a form of persuasive, expressive, or informative writing 
separate type. 



T’s need more practice in developing prompts, etc. for text-based writing. 
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